Autonomous vehicle controlled based upon a lidar data segmentation system

ABSTRACT

An autonomous vehicle is described herein. The autonomous vehicle includes a lidar sensor system. The autonomous vehicle additionally includes a computing system that executes a lidar segmentation system, wherein the lidar segmentation system is configured to identify objects that are in proximity to the autonomous vehicle based upon output of the lidar sensor system. The computing system further includes a deep neural network (DNN), where the lidar segmentation system identifies the objects in proximity to the autonomous vehicle based upon output of the DNN.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/054,088, filed on Aug. 3, 2018, and entitled “AUTONOMOUS VEHICLECONTROLLED BASED UPON A LIDAR DATA SEGMENTATION SYSTEM,” the entirety ofwhich is incorporated herein by reference.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can operate withouthuman conduction. An exemplary autonomous vehicle includes a pluralityof sensor systems, such as, but not limited to, a lidar sensor system, acamera sensor system, and a radar sensor system, amongst others, whereinthe autonomous vehicle operates based upon sensor signals output by thesensor systems.

Conventionally, autonomous vehicles are configured to identify locationsof different objects in a driving environment based upon differentsensor signals. For example, a radar system can identify a range fromthe autonomous vehicle to another vehicle in the driving environment. Inanother example, an object recognition system may be configured toreceive images output by a camera and to identify relative positions ofobjects captured in the images.

In some situations, however, it may be difficult to determine whatobjects are at which locations based upon the sensor signals output bythe sensor systems of the autonomous vehicle. For example, in a lidarpoint cloud representative of the driving environment, it may bedifficult to determine by way of conventional algorithms what points arerepresentative of same objects and what points are representative ofdifferent objects in the driving environment. Conventionally, algorithmsbased upon human-generated rules have been used to perform segmentationof a lidar point cloud to identify objects and their locations in thedriving environment. For example, an algorithm may identify two pointsin a lidar point cloud as belonging to a same object based upon thepoints being within 50 centimeters of one another. This conventionalapproach may not be robust to variations in the driving environment, andcan result in possible errors with respect to object identification bythe autonomous vehicle. For instance, various conventional algorithmsfor performing segmentation of a lidar point cloud may perform poorlywhen certain types of objects are present in a driving environment(e.g., groups of closely-spaced pedestrians, vegetation, non-convexobjects, etc.)

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

Described herein are various technologies pertaining to controllingoperation of an autonomous vehicle. With more specificity, describedherein are various technologies pertaining to segmentation of a lidarpoint cloud, wherein the lidar point cloud is representative ofpositions of objects in the driving environment of the autonomousvehicle at one or more points in time. With still more specificity, alidar data segmentation system is described herein, wherein the lidardata segmentation system comprises a neural network that is trained tooutput data indicative of a probability that a point in a lidar pointcloud is representative of at least one of ground (e.g., a road surface,a sidewalk surface, or other ground surface) or vegetation. The lidardata segmentation system can assign a label to the point based upon theprobability indicated by the neural network output, the label indicatinga classification of a type of object that the point is expected torepresent. The lidar segmentation system then performs a segmentation ofthe lidar point cloud based in part upon the label assigned to thepoint.

In an example, an autonomous vehicle includes a lidar sensor system thatcomprises at least one lidar sensor, and a lidar segmentation system.The lidar sensor system outputs lidar data based on sensor signalsreceived from the at least one lidar sensor, where the lidar data isindicative of positions of objects in the driving environment of theautonomous vehicle at one or more times. In an exemplary embodiment, thelidar data comprises a three-dimensional lidar point cloud, wherein eachpoint in the point cloud indicates a three-dimensional position of anobject or surface of an object in the driving environment of theautonomous vehicle. The lidar sensor system outputs the lidar data tothe lidar segmentation system, whereupon the lidar segmentation systemgenerates a segmentation of the lidar data. By way of example, thesegmentation indicates which points in the lidar point cloud belong tosame objects in the driving environment. Further, the segmentation canindicate a plurality of groups each representative of a differentrespective object in the driving environment, wherein each groupincludes one or more points in the lidar point cloud that arerepresentative of the object.

The lidar segmentation system, in connection with generating thesegmentation of the lidar data, receives the lidar data from the lidarsensor system and identifies various input features with respect to eachof the points represented in the lidar data. By way of example, and notlimitation, input features for a point in a lidar point cloud caninclude distance to a next-closest point in the point cloud, angleformed by the point and adjacent points in the point cloud, mean andvariance of position relative to a local neighborhood of points in thepoint cloud, etc. The lidar segmentation system receives the inputfeatures for each of the points and provides the input features as inputto a deep neural network (DNN). The DNN is trained to output, based uponthe input features for a point, a probability that the point isrepresentative of ground, vegetation, or another type of object. Thelidar segmentation system assigns labels to points in the point cloudbased upon the output of the DNN, the labels indicating a type of objectthat the point is expected to represent. The lidar segmentation systemcan then compute a segmentation of the lidar point cloud based upon thelabels assigned to the points in the lidar point cloud.

In further embodiments, the autonomous vehicle is configured to alignone or more sets of labeled data with one or more sensor signals basedupon the output of the DNN in real time during operation of theautonomous vehicle in the driving environment. In an example, theautonomous vehicle comprises a data alignment system, and a data storagesystem that stores labeled data pertaining to the driving environment(e.g., a semantic map of the driving environment, a height map of thedriving environment, etc.). The data alignment system is configured toalign labeled data stored in the data storage system with the lidar dataoutput by the lidar sensor system. The data alignment system receivesthe output of the DNN and identifies a plurality of points that areindicated in the output of the DNN as having a high probability (e.g.,greater than 90%, greater than 95%, or greater than 99%) of beingrepresentative of ground cover. The data alignment system computes analignment between the labeled data and the points identified as having ahigh probability of being representative of ground cover. A controlsystem of the autonomous vehicle then controls operation of theautonomous vehicle in the driving environment based upon the alignment.By way of example, the control system of the autonomous vehicle candetermine that a direction of travel is unobstructed based jointly uponthe lidar data and the labeled data, wherein joint consideration of thelidar data and the labeled data is based upon the alignment computed bythe data alignment system.

The above summary presents a simplified summary in order to provide abasic understanding of some aspects of the systems and/or methodsdiscussed herein. This summary is not an extensive overview of thesystems and/or methods discussed herein. It is not intended to identifykey/critical elements or to delineate the scope of such systems and/ormethods. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary autonomous vehicle.

FIG. 2 is a functional block diagram of an exemplary object recognitionsystem included within an autonomous vehicle.

FIG. 3A is a top-down view of an exemplary driving environment of anautonomous vehicle.

FIG. 3B is a side view of the exemplary driving environment depicted inFIG. 3A.

FIG. 4 is a functional block diagram of an exemplary system that isconfigured to learn a neural network component included in the objectrecognition system depicted in FIG. 2.

FIG. 5 is a flow diagram illustrating an exemplary methodology forcontrolling operation of a mechanical system of an autonomous vehiclebased upon output of a lidar segmentation system.

FIG. 6 is a flow diagram illustrating an exemplary methodology forlearning a neural network to be included in an object recognition systemof an autonomous vehicle.

FIG. 7 is a flow diagram illustrating an exemplary methodology forcontrolling operation of a mechanical system of an autonomous vehiclebased upon lidar data and a heightmap of a driving environment of theautonomous vehicle.

FIG. 8 is an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to controlling operation of anautonomous vehicle based upon lidar segmentation data generated throughuse of a DNN are now described with reference to the drawings, whereinlike reference numerals are used to refer to like elements throughout.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of one or more aspects. It may be evident, however, thatsuch aspect(s) may be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate describing one or more aspects. Further, itis to be understood that functionality that is described as beingcarried out by certain system components may be performed by multiplecomponents. Similarly, for instance, a component may be configured toperform functionality that is described as being carried out by multiplecomponents.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Further, as used herein, the terms “component” and “system” are intendedto encompass computer-readable data storage that is configured withcomputer-executable instructions that cause certain functionality to beperformed when executed by a processor. The computer-executableinstructions may include a routine, a function, or the like. It is alsoto be understood that a component or system may be localized on a singledevice or distributed across several devices. Further, as used herein,the term “exemplary” is intended to mean serving as an illustration orexample of something and is not intended to indicate a preference.

With reference now to FIG. 1, an exemplary autonomous vehicle 100 isillustrated. The autonomous vehicle 100 can navigate about roadwayswithout human conduction based upon sensor signals output by sensorsystems of the autonomous vehicle 100. The autonomous vehicle 100includes a lidar sensor system 101 and a plurality of additional sensorsystems 102-104 (a second sensor system 102 through an Nth sensor system104). The sensor systems 102-104 may be of different types and arearranged about the autonomous vehicle 100. For example, the secondsensor system 102 may be a radar sensor system and the Nth sensor system104 may be a camera (image) system. Other exemplary sensor systemsinclude GPS sensor systems, sonar sensor systems, infrared sensorsystems, and the like.

The autonomous vehicle 100 further includes several mechanical systemsthat are used to effectuate appropriate motion of the autonomous vehicle100. For instance, the mechanical systems can include but are notlimited to, an engine 106, a braking system 108, and a steering system110. The engine 106 may be an electric engine or a combustion engine.The braking system 108 can include an engine brake, brake pads,actuators, a regenerative braking system, and/or any other suitablecomponentry that is configured to assist in decelerating the autonomousvehicle 100. The steering system 110 includes suitable componentry thatis configured to control the direction of movement of the autonomousvehicle 100.

The autonomous vehicle 100 additionally comprises a computing system 112that is in communication with the sensor systems 101-104 and is furtherin communication with the engine 106, the braking system 108, and thesteering system 110. The computing system 112 includes a processor 114and memory 116 that includes computer-executable instructions that areexecuted by the processor 114. In an example, the processor 114 can beor include a graphics processing unit (GPU), a plurality of GPUs, acentral processing unit (CPU), a plurality of CPUs, anapplication-specific integrated circuit (ASIC), a microcontroller, aprogrammable logic controller (PLC), a field programmable gate array(FPGA), or the like.

The memory 116 comprises an object recognition system 118 that isconfigured to identify objects (in proximity to the autonomous vehicle100) captured in sensor signals output by the sensor systems 101-104. Aswill be described in greater detail below (and referring briefly to FIG.2), the object recognition system 118 includes a lidar segmentationsystem 202, and may further include a plurality of additional objectrecognition components 204-206 (a second objection recognition component204 through an Mth objection recognition component 206), wherein thelidar segmentation system 202 and the objection recognition components204-206 are configured to output data indicative of positions and/orpredefined types of objects in a driving environment of the autonomousvehicle 100. These predefined types of objects can include, but are notlimited to, pedestrian, bike, car, truck, bus, ground cover (e.g., pavedroadways, dirt, grass) etc. Each of the object recognizer components202-206 independently generates output based upon at least one sensorsignal. For instance, the lidar segmentation system 202 outputs dataindicative of positions of objects in the driving environment of theautonomous vehicle 100 based upon lidar data output by the lidar sensorsystem 101. In further examples, the second object recognizer component204 outputs data indicative of types of objects based upon images outputby a camera, the Mth object recognizer component 206 outputs dataindicative of ranges to objects based upon radar signals, etc.

The memory 118 additionally includes a control system 120 that isconfigured to receive output of the object recognition system 118, andis further configured to control at least one of the mechanical systemsof the autonomous vehicle 100 (the engine 106, the braking system 108,and/or the steering system 110) based upon the output of the objectrecognition system 118.

Exemplary operation of the autonomous vehicle 100 is now set forth.FIGS. 3A and 3B depict different views of an exemplary drivingenvironment 300 in which the autonomous vehicle 100 operates. Withreference now to FIG. 3A, the autonomous vehicle 100 is depicted in thedriving environment 300, wherein the driving environment includes afirst vehicle 302, two pedestrians 304, 306 standing to the right of thevehicle 302, and a bush 308 that stands to the right of the pedestrians304, 306. The autonomous vehicle 100 emits a laser beam 310 into thedriving environment 300 (e.g., by way of the lidar sensor system 101).The laser beam 310 is scanned 360° around the vehicle 100, and impingesupon the various objects 302-308 in the environment 300. The lidarsensor system 101 receives reflections of the beam 310, and based uponsuch reflections outputs data indicative of positions of a plurality ofpoints of intersection 312 of the beam 310 with the objects 302-308 inthe driving environment 300. By way of example, the lidar sensor system101 outputs data indicating a three-dimensional position of each of thepoints 312 (e.g., relative to the autonomous vehicle 100, relative to apoint of reference in the driving environment 300 or outside of thedriving environment 300, etc.).

It is to be understood that while a single laser beam 310 and aplurality of points of intersection 312 of the beam 310 with objects302-308 are depicted in FIG. 3A, the lidar sensor system 101 can emit aplurality of laser beams into a driving environment of the autonomousvehicle 100. For instance, and referring now to FIG. 3B, a side view ofthe objects 302-308 in the driving environment 300 is depicted. Inexemplary embodiments, the lidar sensor system 101 emits a plurality oflaser beams into the driving environment of the vehicle 100, whereineach of the beams has a different angular alignment with respect to ahorizontal reference plane of the vehicle 100. Further, each of aplurality of sensors in the lidar sensor system 101 may be aligned witha different respective lidar beam emitted by the lidar sensor system101, such that the lidar sensor system 101 outputs data indicative of aplurality of points of intersection for each of the plurality of beamsover a period of time. As depicted in FIG. 3B, the plurality of laserbeams scan across the objects 302-308 in the driving environment 300 andintersect the objects along scan lines 316-320. Each of the points 312where a laser beam of the lidar sensor system 101 intersects one of theobjects 302-308 in the driving environment lies along one of the scanlines 316-322. Hence, each of the plurality of lidar sensors in thelidar sensor system 101 outputs data indicative of a point ofintersection of a respective laser beam with various objects in thedriving environment 300 along a respective scan line. A lidar pointcloud output by the lidar sensor system 101 can therefore be considereda rastered image of objects in the driving environment, comprising asmany scan lines as lasers are included in the lidar sensor system 101.It is to be appreciated that over the period of a 360° scan of a drivingenvironment of the autonomous vehicle 100, the lidar sensor system 101can output data indicative of positions of a larger number of pointsthan are depicted in FIGS. 3A and 3B (e.g., tens of thousands of points,hundreds of thousands of points, millions of points, etc.). Furthermore,it is to be appreciated that points of intersection of laser beams withground cover may be represented in a lidar point cloud output by thelidar sensor system 101. For example, the scan line 322 liessubstantially along a ground surface 324, and the points of intersection312 lying along the scan line 322 represent intersections of a laserbeam of the lidar sensor system 101 with the ground surface 324.

For many objects in a driving environment of an autonomous vehicle, thepoints of intersection of laser beams emitted by the lidar sensor system101 are relatively regularly arranged along surfaces of the object. Forinstance, and referring again to FIG. 3A, the points of intersection 312of the beam 310 with the vehicle 302 are regularly arranged alongsurfaces of the vehicle (e.g., doors, side panels, etc.). However, forsome objects, the points of intersection with the laser beams areirregularly positioned. For instance, the points of intersection 312 inthe driving environment include points of intersection 314 with the bush308. The points of intersection 314 may not be regularly spaced aroundapparent boundaries of the bush 308, for example due to movement ofbranches and leaves of the bush 308 in wind, irregular positioning ofbranches and leaves of the bush 308, etc. These irregularitiesassociated with vegetation in a driving environment of an autonomousvehicle can interfere with conventional lidar point cloud segmentationsystems that identify objects in a lidar point cloud. By way of example,a conventional lidar point cloud segmentation system operating in thedriving environment 300 may determine that the points of intersectionwith the bush 308 represent multiple objects rather than a singleobject. In another example, a conventional lidar point cloudsegmentation system may determine that points of intersection with thebush 308 represent the same object as points of intersectionrepresentative of the pedestrian 306

Referring again to FIG. 2, various details pertaining to the lidarsegmentation system 202 are now described. The lidar segmentation system202 is configured to identify points in a lidar point cloud that arelikely to be representative of vegetation and/or ground cover, and toperform a segmentation of the lidar point cloud that takes appropriateconsideration of such objects. For example, the lidar segmentationsystem 202 can be configured to identify adjacent points that are likelyto be representative of vegetation, and to consider these points to berepresentative of a same object (e.g., a single bush). In anotherexample, the lidar segmentation system 202 can be configured to identifypoints that are likely to be representative of ground cover, and toseparate these points from adjacent points in the lidar point cloud thatare not likely to be representative of ground cover. Hence, the lidarsegmentation system 202 can be prevented from erroneously determiningthat non-ground objects are part of ground cover in the drivingenvironment of the autonomous vehicle 100.

The lidar segmentation system 202 further comprises a lidarpreprocessing component 208, a neural network component 210, and a lidarpostprocessing component 212. The lidar segmentation system 202 receiveslidar data from the lidar sensor system 101, wherein the lidar data isindicative of positions of surfaces of a plurality of objects in adriving environment of the autonomous vehicle 100. The lidarpreprocessing component 208 performs various processing operations overthe lidar data and outputs a plurality of input features to the neuralnetwork component 210. The neural network component 210 is configured tooutput a probability that a point is representative of at least one ofground cover or vegetation based upon the input features associated withthe point. The lidar postprocessing component 212 then performs asegmentation of the lidar point cloud based upon the probability outputby the neural network component 210.

Exemplary operations of the lidar segmentation system 202 are now setforth. The lidar segmentation system 202 receives a lidar point cloudfrom the lidar sensor system 101, wherein the point cloud is indicativeof positions of points on surfaces of the objects 302-308 (and otherobjects that may be present) in the driving environment 300 of thevehicle 100 (e.g., as described above with respect to FIGS. 3A and 3B).The lidar preprocessing component 208 performs an analysis over thepoints in the point cloud to identify various features relative to eachof the points in the point cloud. By way of example, the lidarpreprocessing component 208 parameterizes the point cloud according to atraining scheme used to learn the neural network component 210. Forinstance, the lidar preprocessing component 208 can compute, for each ofthe points in the point cloud, a distance to a next-closest point in thepoint cloud, a distance from the point to the autonomous vehicle 100, anumber of points within a threshold distance of the point, etc., wheresuch features are provided as input to a DNN during training of the DNN(e.g., as described below with respect to FIG. 4). The preprocessorcomponent 208 then provides these features as input features to theneural network component 210 for each of the points in the point cloud.

The neural network component 210 comprises a DNN that has a plurality oflayers including an input layer, a plurality of hidden layers (e.g.,greater than 5 layers, greater than 10 layers, greater than 50 layers,etc.), and an output layer. Responsive to receipt of input features fora point in the lidar point cloud at the input layer of the DNN, theinput features are propagated through the hidden layers of the DNN. TheDNN is configured to output, for each of the points in the point cloud,one or more probabilities, each probability indicating a likelihood thatthe point is representative of an object of a certain type. By way ofexample, the DNN can be configured such that the output layer is asoftmax layer that outputs a first probability that the input point(e.g., the point represented by the input features provided at the inputlayer of the DNN) represents a first type of object, a secondprobability that the input point represents a second type of object,etc. through an nth probability that the input points represents an nthtype of object, where then probabilities sum to unity. In an exemplaryembodiment, the output layer outputs a first probability that the inputpoint is representative of vegetation, a second probability that theinput point is representative of ground cover, and a third probabilitythat the input point is representative of any other type of object(e.g., cars, trucks, pedestrians, bicycles, etc.). As discussed below ingreater detail with respect to FIG. 4, the DNN is learned based uponlabeled training data (e.g., labeled lidar point cloud data) such thatthe output of the DNN indicates a high probability (e.g., greater than80%, greater than 90%, greater than 95%, etc.) that the input point isground cover when the input point is actually representative of groundcover, a high probability that the input point is vegetation when theinput point is actually representative of vegetation, etc.

The lidar post processing component 212 receives the output of the DNN(e.g., the n probabilities indicated by the output layer of the DNN)from the neural network component 210 and outputs, based upon the outputof the DNN, a segmentation of the lidar point cloud (e.g., as receivedfrom the lidar sensor system 101) that indicates which points in thepoint cloud are representative of same objects in the drivingenvironment of the autonomous vehicle 100. In an exemplary embodiment,the lidar post processing component 212 performs a segmentation of thelidar point cloud by excluding points from consideration that areindicated by the output of the DNN as being likely to be representativeof vegetation and/or ground. Stated differently, the lidar postprocessing component 212 performs a segmentation of only those points inthe point cloud that are indicated by the output of the DNN as not beinglikely to represent ground or vegetation. By excluding pointsrepresentative of ground and vegetation from consideration by asegmentation algorithm, the lidar post processing component 212 canreduce errors associated with poor performance of the segmentationalgorithm in identifying ground and vegetation features. For instance,excluding points representative of ground cover from consideration by asegmentation algorithm can reduce errors caused by an inability of thealgorithm to distinguish between the ground and objects in contact withthe ground (e.g., pedestrians' feet, car tires, or other object incontact with the ground). In another example, excluding pointsrepresentative of vegetation from consideration by a segmentationalgorithm can reduce errors caused by an inability of the algorithm toidentify that scattered points in the point cloud are representative ofa same vegetative object, such as a bush, a leafy tree branch, etc.

The lidar post processing component 212 identifies a first group ofpoints in the point cloud that are indicated by the output of the DNN asbeing likely to represent ground, and/or a second group of points in thepoint cloud that are indicated by the output of the DNN as being likelyto represent vegetation. In an embodiment, the lidar post processingcomponent 212 can identify points that are likely to be representativeof ground and/or vegetation based upon the output of the DNN and aprobability threshold. For instance, the lidar post processing component212 can label a point as being representative of ground based upon theoutput of the DNN indicating that the point has a probability of beingground that is above the probability threshold. In various examples, theprobability threshold may be 75%, 90%, 95%, or other desirablethreshold. Responsive to identifying a point as being likely to berepresentative of ground or vegetation (e.g., based upon the probabilitythreshold), the lidar post processing component 212 assigns a label tothe point that indicates ground or vegetation, accordingly.

Upon determining which points in the lidar point cloud are likely to berepresentative of ground and/or vegetation, the lidar post processingcomponent 212 generates a segmentation of points in the point cloud thatare not labeled as ground or vegetation. The segmentation comprises dataindicative of which points are representative of which objects in thedriving environment of the autonomous vehicle 100. By way of an example,generating the segmentation comprises assigning respective labels toeach of the points in the point cloud, wherein each of the labelsindicates a group to which its point belongs, each group beingrepresentative of a different object in the driving environment. Thelidar post processing component 212 can generate the segmentationaccording to any suitable segmentation algorithm. In an example, thelidar post processing component 212 generates the segmentation basedupon output of a neural network that is trained to output dataindicative of whether points in a lidar point cloud are representativeof same objects. In another example, the segmentation is generated basedupon an algorithm that considers distance between points in the pointcloud, relative angles formed by points in the point cloud, convex hullsof subsets of points in the point cloud, etc. Other approaches togenerating the segmentation of the points not labeled vegetation orground are also contemplated.

Responsive to generating the segmentation, the lidar post processingcomponent 212 outputs the segmentation to the object recognition system118. The object recognition system 118 outputs data indicative of one ormore objects in the driving environment to the control system 120 basedupon the segmentation generated by the lidar segmentation system 202. Byway of example, the object recognition system 118 outputs dataindicative of positions and/or types of objects in the drivingenvironment. In exemplary embodiments, the object recognition system 118outputs data indicative of positions and/or types of objects in thedriving environment of the autonomous vehicle 100 based upon thesegmentation output by the lidar segmentation system 202 and output ofone or more of the additional object recognizer components 204-206.Subsequently, the control system 120 of the vehicle 100 can outputcontrol signals to at least one of the engine 106, the braking system108, or the steering system 110 based upon the output of the objectrecognition system 118, such that the autonomous vehicle is controlledbased at least in part upon the segmentation of the lidar point cloudgenerated by the lidar segmentation system 202.

In various embodiments, it may be desirable to control operation of anautonomous vehicle based at least in part on labeled data that pertainsto a potential operating environment of the vehicle (e.g., a city orother region in which the autonomous vehicle is to drive). For instance,it may be desirable to control operation of an autonomous vehicle basedupon a semantic map that includes data pertaining to route information(e.g., locations of streets, directions of traffic on streets, roadclosures, etc.) and locations of interest (e.g., restaurants,fueling/charging stations, entertainment venues, etc.). In anotherexample, it may be desirable to control operation of an autonomousvehicle based upon a heightmap of the driving environment of theautonomous vehicle.

Referring once again to FIG. 1, the computing system 112 of theautonomous vehicle 100 further includes a data store 122 that storeslabeled data 124. The labeled data 124 includes data pertaining to apotential operating environment of the autonomous vehicle 100 (e.g., anarea that includes the surroundings of the vehicle 100 that make up thedriving environment of the vehicle 100). The labeled data 124 isgenerated independent of the sensor systems 101-104 of the autonomousvehicle 100 and is labeled based upon ground truth data pertaining tothe potential operating environment (e.g., based upon human input). Thelabeled data 124 includes a semantic map 126 of a potential operatingenvironment of the autonomous vehicle 100 (e.g., an area that includesthe surroundings of the vehicle 100 that make up the driving environmentof the vehicle 100) and a heightmap 128 of the potential operatingenvironment. The heightmap 128 comprises data indicating a height foreach of a plurality of points in a two-dimensional space that comprisesan overhead view of the potential operating environment of theautonomous vehicle 100. In an exemplary embodiment, the autonomousvehicle 100 can identify a fastest available route to a destinationbased upon the semantic map 126.

It may further be desirable to control operation of the autonomousvehicle 100 based jointly upon the labeled data 124 and the sensorsignals output by the sensor systems 101-104 of the autonomous vehicle100. For example, it may be desirable that the autonomous vehicle 100identify that sensor data is representative of a same object in thedriving environment that is also represented in the labeled data, andperform one or more functions based upon the sensor data and the labeleddata 124 being representative of the same object. In order to controloperation of the autonomous vehicle 100 based jointly upon sensor dataand the labeled data 124, the sensor data and the labeled data 124 arealigned to ensure that sensor signals are accurately matched tocorresponding information in the labeled data 124. In an example, thelidar data output by the lidar sensor system 101 is aligned with theheight map 128. The lidar data and the height map 128 are aligned suchthat the computing system 112 can readily determine that first data inthe height map 128 is representative of a first location of a firstpoint in the lidar data. By way of example, the computing system 112 cancompute a coordinate transform from a coordinate system of the lidarpoint cloud to a coordinate system of the height map 128 (andvice-versa). Hence, the computing system 112 can match coordinates of apoint in the point cloud to coordinates in the height map 128 todetermine a ground height at a position of the point.

Alignment between sensor data and the labeled data 124 can be initiallyachieved during calibration of systems of the autonomous vehicle 100prior to operation of the autonomous vehicle 100 in the drivingenvironment. By way of example, in a calibration environment the lidarsensor system 101 generates a lidar point cloud, wherein the lidar pointcloud is representative of a known geometry of the calibrationenvironment. The computing system 112 of the vehicle 100 can compute analignment between labeled data 124 and the lidar point cloud based uponthe geometry of the calibration environment being known. However, theautonomous vehicle 100 is subject to misalignment of the sensor data andthe labeled data 124 over time as the autonomous vehicle 100 operates ina driving environment. Conventionally, in order to maintain alignment ofsensor data and labeled data, the autonomous vehicle must beperiodically recalibrated in the environment having the known geometry,requiring downtime of the autonomous vehicle and removal of theautonomous vehicle from the driving environment.

In exemplary embodiments, the autonomous vehicle 100 is configured toalign sensor data from one or more of the sensor systems 101-104 withlabeled data 124 in real-time during operation of the vehicle 100 in adriving environment. Referring now once again to FIG. 2, the lidarsegmentation system 202 further comprises a data alignment system 214that computes an alignment between lidar data output by the lidar sensorsystem 101 and the labeled data 124 based upon output of the neuralnetwork component 210. In an exemplary embodiment, the data alignmentsystem 214 receives output of the DNN from the neural network component210. The data alignment system 214 identifies points in the point cloudfor which the output of the DNN indicates a high probability (e.g.,greater than 90% probability, greater than 95% probability, greater than99% probability) that the point is representative of ground. The dataalignment system 214 then computes an alignment between the coordinatesystem of the lidar point cloud and the coordinate system of theheightmap 128 based upon the identified points and the heightmap 128. Ingeneral, the alignment computed by the data alignment system 214 basedupon the identified points is more accurate than an alignment computedbased upon a group of points in the lidar point cloud that also includespoints that are not representative of ground.

Responsive to receipt of alignment data from the data alignment system214, the lidar post processing component 212 can generate a segmentationof the lidar data based upon the labeled data 124 or update an existingsegmentation of the lidar data based upon the labeled data 124. In anexample, the lidar post processing component 212 receives a coordinatetransform from the data alignment system 214. The coordinate transformindicates a transformation from a coordinate system of the lidar data toa coordinate system of at least one dataset in the labeled data 124. Thelidar post processing component 212 employs the coordinate transform toupdate a segmentation based upon characteristics of points in the lidarpoint cloud that are indicated in the labeled data 124.

For example, the lidar post processing component 212 can identify apoint in the lidar point cloud for which the output of the DNN indicatesa moderate probability (e.g., 40%-80%, 50%-70%, or 55%-65%) of the pointbeing representative of ground cover. For instance, referring againbriefly to FIG. 3B, a point 326 near the bottom of a tire of the vehicle302 in the driving environment 300 can be indicated in the output of theDNN as having a probability of 55% of being representative of groundcover (e.g., due in part to its proximity to the ground surface 324).The lidar post processing component 212 can compute, based upon thecoordinate transform, coordinates of the identified point in theheightmap 128. Based upon a height of the identified point indicated inthe heightmap 128, the lidar post processing component 212 can determinewhether the identified point is or is not representative of ground. Forexample, if the height of the point as indicated in the lidar pointcloud is significantly greater than the height indicated in theheightmap 128 (e.g., 20 centimeters greater, 10 centimeters greater, 5centimeters greater), the lidar post processing component 212 can assigna label to the point indicating that the point is representative of anobject other than ground. In another example, if the height of the pointas indicated in the lidar point cloud is substantially similar to theheight indicated in the heightmap 128 (e.g., within 20 centimeters,within 10 centimeters, within 5 centimeters), the lidar post processingcomponent 212 can assign a label to the point indicating that the pointis representative of ground cover. It is to be understood that there maybe uncertainty associated with heights indicated in the heightmap data128. In exemplary embodiments, the heightmap 128 includes dataindicative of an uncertainty threshold (e.g., a range of potentialheights) for one or more of the heights indicated in the heightmap 128.The lidar post processing component 212 can assign a label to a point inthe lidar point cloud based upon the uncertainty threshold, the labelindicating that the point is representative of ground cover. Forinstance, if a height of a point as indicated by the lidar point cloudis within a range of potential heights indicated in the heightmap 128,the lidar post processing component 212 assigns a label to the pointindicating that the point is representative of ground.

It is to be understood that the lidar post processing component 212 canassign the labels to points in the lidar point cloud based upon thealignment, as described above, prior to performing a segmentation of thelidar point cloud. In an exemplary embodiment, the data alignment system214 receives the output of the DNN from the neural network component210, and computes the alignment as described above. The lidar postprocessing component 212, based upon the alignment and the heightmap128, evaluates points in the lidar point cloud that potentiallyrepresent ground cover, and assigns labels to those points with heightsconsistent with the heightmap 128. The lidar post processing component212 can then generate the segmentation of the lidar point cloud basedupon the labeled points being representative of ground.

By way of example, and not limitation, the lidar post processingcomponent 212 can evaluate points in the lidar point cloud that areindicated by the output of the DNN as having a probability ofrepresenting ground cover that is above a threshold amount (e.g.,greater than 30%, greater than 40%, greater than 50%) against heightsindicated in the heightmap 128. To each point indicated by the lidarpoint cloud as having a height within a range of potential heightsindicated in the heightmap 128, the lidar post processing component 212assigns a label indicating that the point is representative of ground.The lidar post processing component 212 can then perform a segmentationof the remaining unlabeled points in the lidar point cloud based uponany suitable segmentation algorithm. The control system 120 cansubsequently output control signals to at least one of the engine 106,the braking system 108, or the steering system 110 to control operationof the autonomous vehicle 100 based upon the segmentation of the lidarpoint cloud.

With reference now to FIG. 4, an exemplary computing system 400 that isconfigured to learn the neural network component 210 is illustrated. Thecomputing system 400 includes a processor 402 and memory 404, whereinthe memory 404 includes instructions that are executed by the processor402. The computing system 400 additionally includes a data store 406that comprises labeled training data 408. The memory 404 includes aneural network learner 410 that learns the neural network component 210.The neural network learner 410 receives the labeled training data 408,wherein the labeled training data 408 comprises lidar point cloud data(e.g., parameterized as described above with respect to input featuresof the DNN), wherein the lidar point cloud data further includes groundtruth data indicating which points are representative of vegetation, andwhich points are representative of ground in the lidar point cloud data.The neural network learner 410 learns the neural network component 210such that the neural network component 210 outputs a high groundprobability for points in the point cloud that are representative ofground, a high vegetation probability for points representative ofvegetation, and a low probability of vegetation and ground for pointsrepresentative of neither vegetation nor ground. While not shown, theneural network component 210 can be validated based upon a test trainingdata set, which is also labeled, to ensure that the neural networkcomponent 210 produces outputs with acceptable characteristics (e.g.,such that the neural network component 210 outputs a high vegetationprobability for points representative of vegetation and a low vegetationprobability for points that are not representative of vegetation).

FIGS. 5-7 illustrate exemplary methodologies relating to controlling anautonomous vehicle based upon data output by a lidar data segmentationsystem. While the methodologies are shown and described as being aseries of acts that are performed in a sequence, it is to be understoodand appreciated that the methodologies are not limited by the order ofthe sequence. For example, some acts can occur in a different order thanwhat is described herein. In addition, an act can occur concurrentlywith another act. Further, in some instances, not all acts may berequired to implement a methodology described herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

Referring now to FIG. 5, an exemplary methodology 500 for controllingoperation of an autonomous vehicle is illustrated. The methodology 500starts at 502, and at 504 lidar data is received, wherein the lidar datacomprises a plurality of points representative of positions of objectsin a driving environment of an autonomous vehicle. At 506, a label isassigned to a first point in the points based upon output of a DNN. Byway of example, responsive to receiving input features pertaining to thefirst point, the DNN outputs respective probabilities that the firstpoint is representative of ground cover, vegetation, or another type ofobject. In a non-limiting example, the label is assigned to the firstpoint responsive to determining that a probability of the first pointbeing either vegetation or ground as indicated in the output of the DNNis greater than a pre-defined threshold value. The label assigned to thefirst point indicates that the first point is representative of groundcover or vegetation in the driving environment. At 508, a segmentationof the lidar data is generated based upon the first point being excludedfrom the segmentation. By generating the segmentation based uponexcluding the first point from the segmentation, errors associated withpoor performance of a segmentation algorithm relative to ground andvegetation features are avoided. At 510, at least one of an engine, abraking system, or a steering system of the autonomous vehicle iscontrolled based upon the segmentation generated at 508. For example, anobject recognition system of the autonomous vehicle can generate aprediction of future behavior of one or more objects in the drivingenvironment based upon the segmentation, and can control steering of theautonomous vehicle based upon such prediction. The methodology 500completes at 512.

Turning now to FIG. 6, an exemplary methodology 600 that facilitateslearning a DNN is illustrated, wherein the DNN is configured for use ina lidar segmentation system of an autonomous vehicle. The methodology600 starts at 602, and at 604 labeled training data is received, whereinthe training data comprises labeled lidar point cloud data. At 606, aDNN is learned based upon the training data. The DNN is configured suchthat responsive to receiving a point in a lidar point cloud as input(e.g., when the DNN receives a vector of input features representativeof the point at an input layer of the DNN), the DNN outputs a pluralityof probabilities, each probability indicative of a likelihood that thepoint represents a different respective type of object. For instance,responsive to receiving a point in a lidar point cloud as input, the DNNoutputs a first probability that the point is representative ofvegetation, a second probability that the point is representative ofground, and a third probability that the point is representative of someother type of object, where the three probabilities sum to unity. TheDNN is learned at 606 such that the output of the DNN indicates a highprobability that the point is representative of vegetation when theinput point is a point representative of vegetation, the output of theDNN indicates a high probability that the point is representative ofground when the input point is a point representative of ground, and theoutput of the DNN indicates a high probability that the point isrepresentative of some other type of object when the input point is apoint representative of neither ground nor vegetation. The methodology600 completes at 608.

Referring now to FIG. 7, an exemplary methodology 700 that facilitatescontrolling operation of an autonomous vehicle based upon sensor dataand labeled data is illustrated. The methodology 700 begins at 702 andat 704, lidar data is received, wherein the lidar data comprises aplurality of points representative of positions of objects in a drivingenvironment of an autonomous vehicle. At 706, an alignment is computedbetween the lidar data and a heightmap based upon a first point in thepoints, wherein the first point is selected based upon output of a DNNindicating that the first point is representative of ground. At 708, atleast one of an engine, a braking system, or a steering system of theautonomous vehicle is controlled based upon the lidar data and theheightmap, wherein the alignment computed at 706 facilitates jointconsideration of the lidar data and the heightmap. By way of example, asegmentation of the lidar data can be generated based at least in partupon the heightmap being aligned with the lidar data. The methodology700 ends at 710.

Referring now to FIG. 8, a high-level illustration of an exemplarycomputing device 800 that can be used in accordance with the systems andmethodologies disclosed herein is illustrated. For instance, thecomputing device 800 may be or include the computing system 112 or thecomputing system 400. The computing device 800 includes at least oneprocessor 802 that executes instructions that are stored in a memory804. The instructions may be, for instance, instructions forimplementing functionality described as being carried out by one or moremodules and systems discussed above or instructions for implementing oneor more of the methods described above. The processor 802 may be a GPU,a plurality of GPUs, a CPU, a plurality of CPUs, a multi-core processor,etc. The processor 802 may access the memory 804 by way of a system bus806. In addition to storing executable instructions, the memory 804 mayalso store point cloud data, vectors of output features for points in apoint cloud, segmentation data, training data, etc.

The computing device 800 additionally includes a data store 808 that isaccessible by the processor 802 by way of the system bus 806. The datastore 808 may include executable instructions, sensor data, outputfeature vector data, training data, segmentation data, etc. Thecomputing device 800 also includes an input interface 810 that allowsexternal devices to communicate with the computing device 800. Forinstance, the input interface 810 may be used to receive instructionsfrom an external computer device, etc. The computing device 800 alsoincludes an output interface 812 that interfaces the computing device800 with one or more external devices. For example, the computing device800 may transmit control signals to the engine 106, the braking system108, and/or the steering system 110 by way of the output interface 812.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 800 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 800.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproducedata magnetically and discs usually reproduce data optically withlasers. Further, a propagated signal is not included within the scope ofcomputer-readable storage media. Computer-readable media also includescommunication media including any medium that facilitates transfer of acomputer program from one place to another. A connection, for instance,can be a communication medium. For example, if the software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio and microwave are includedin the definition of communication medium. Combinations of the aboveshould also be included within the scope of computer-readable media.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Program-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the detailed description or theclaims, such term is intended to be inclusive in a manner similar to theterm “comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. An autonomous vehicle (AV) comprising: an engine;a braking system; a steering system; a lidar sensor; and a computingsystem that is in communication with the engine, the braking system, thesteering system, and the lidar sensor, wherein the computing systemcomprises: a processor; and memory that stores instructions that, whenexecuted by the processor, cause the processor to perform actscomprising: receiving lidar data, the lidar data based upon output ofthe lidar sensor, the lidar data comprising a plurality of pointsrepresentative of positions of objects in a driving environment of theAV; assigning a label to a first point in the points that indicates thatthe first point is representative of ground cover or vegetation basedupon output of a deep neural network (DNN) that is configured toclassify points as being representative of ground cover or vegetation;generating a segmentation of the lidar data based upon the label beingassigned to the first point, wherein the segmentation is indicative of asecond point in the lidar data and a third point in the lidar data beingrepresentative of a same object; and controlling at least one of theengine, the braking system, or the steering system during operation ofthe AV in the driving environment based upon the segmentation.
 2. The AVof claim 1, wherein the output of the DNN comprises a probability thatthe first point is representative of vegetation.
 3. The AV of claim 2,wherein the label indicates that the first point is representative ofvegetation, wherein assigning the label is based upon the probabilityexceeding a threshold value.
 4. The AV of claim 1, wherein responsive toreceipt of input features pertaining to the first point, the output ofthe DNN comprises: a first probability that the first point isrepresentative of vegetation; a second probability that the first pointis representative of ground cover; and a third probability that thefirst point is representative of an object of a type other thanvegetation or ground cover.
 5. The AV of claim 1, wherein the output ofthe DNN comprises a probability that the first point is representativeof ground cover.
 6. The AV of claim 1, wherein generating thesegmentation of the lidar data comprises assigning group labels to thepoints in the lidar data, each group label indicating one of a pluralityof groups of points, each group of points representative of a differentrespective object in the driving environment.
 7. The AV of claim 6,wherein generating the segmentation comprises assigning a same firstgroup label to the first point and a fourth point based upon the firstpoint and the fourth point being labeled as representative ofvegetation, the first group label indicative of first group that isrepresentative of a vegetation object in the driving environment.
 8. TheAV of claim 6, wherein generating the segmentation comprises assigningdifferent group labels to the first point and a fourth point based uponthe first point being labeled as representative of vegetation and thefourth point not being labeled as representative of vegetation.
 9. TheAV of claim 1, wherein generating the segmentation comprises excludingthe first point from consideration by a segmentation algorithm basedupon the label being assigned to the first point.
 10. The AV of claim 1,the acts further comprising assigning a respective label to each of afirst group of points in the points based upon output of the DNN, thelabels assigned to the first group of points indicating that the firstgroup of points are representative of vegetation or ground cover in thedriving environment, wherein generating the segmentation is basedfurther upon the labels assigned to the first group of points.
 11. TheAV of claim 10, wherein generating the segmentation comprises excludingthe first point and the first group of points from consideration by asegmentation algorithm based upon the labels being assigned to the firstpoint and the first group of points.
 12. A method for controllingoperation of an autonomous vehicle (AV), comprising: receiving lidardata from a lidar sensor system of the AV, the lidar data based uponoutput of at least one lidar sensor, the lidar data comprising aplurality of points representative of positions of objects in a drivingenvironment of the AV; assigning a label to a first point in the pointsbased upon output of a deep neural network (DNN) that is configured tooutput a probability that a point in lidar data is representative ofground cover or vegetation, the label indicating that the first point isrepresentative of ground cover or vegetation in the driving environment;generating a segmentation of the lidar data based upon the label beingassigned to the first point, wherein the segmentation is indicative of asecond point in the lidar data and a third point in the lidar data beingrepresentative of a same object; and controlling, based upon thesegmentation, at least one of an engine of the AV, a braking system ofthe AV, or a steering system of the AV during operation of the AV in thedriving environment.
 13. The method of claim 12, wherein the output ofthe DNN comprises a probability that the first point is representativeof vegetation.
 14. The method of claim 12, wherein the output of the DNNcomprises a probability that the first point is representative of groundcover.
 15. The method of claim 14, wherein the label indicates that thefirst point is representative of ground cover, wherein assigning thelabel is based upon the probability exceeding a threshold value.
 16. Themethod of claim 12, wherein generating the segmentation of the lidardata comprises assigning group labels to the points in the lidar data,each group label indicating one of a plurality of groups of points, eachgroup of points representative of a different respective object in thedriving environment.
 17. The method of claim 12, wherein generating thesegmentation comprises executing a lidar segmentation algorithm over thelidar data based upon the label being assigned to the first point. 18.The method of claim 12, the acts further comprising assigning arespective label to each of a first group of points in the points basedupon output of the DNN, the labels assigned to the first group of pointsindicating that the first group of points are representative ofvegetation or ground cover in the driving environment, whereingenerating the segmentation is based further upon the labels assigned tothe first group of points.
 19. The method of claim 18, whereingenerating the segmentation comprises excluding the first point and thefirst group of points from consideration by a segmentation algorithmbased upon the labels being assigned to the first point and the firstgroup of points.
 20. An autonomous vehicle (AV) comprising: acomputer-readable storage medium comprising instructions that, whenexecuted by a processor, cause the processor to perform acts comprising:receiving a lidar point cloud from a lidar sensor system of the AV, thelidar point cloud based upon output of at least one lidar sensor, thelidar point cloud comprising a plurality of points representative ofpositions of objects in a driving environment of the AV; assigning alabel to a first point in the points that indicates that the first pointis representative of vegetation in the driving environment based uponoutput of a deep neural network (DNN), wherein the DNN is trained toreceive features pertaining to the first point as input and to output aprobability that the first point is representative of vegetation in thedriving environment, the label assigned to the first point based uponthe probability exceeding a threshold probability; generating asegmentation of the lidar point cloud based upon the label beingassigned to the first point, wherein the segmentation indicates that asecond point in the points and a third point in the points arerepresentative of a same object; and controlling, based upon the secondpoint and the third point being indicated as representative of the sameobject, at least one of an engine of the AV, a braking system of the AV,or a steering system of the AV during operation of the AV in the drivingenvironment.